security: cache derived SCRAM stored key by nguyen-andrew · Pull Request #31054 · redpanda-data/redpanda

nguyen-andrew · 2026-07-08T21:26:47Z

HTTP Basic auth (PandaProxy, Schema Registry, Admin API) and Kafka SASL/PLAIN
rederive the credential's stored SCRAM key on every request: a PBKDF2-style
loop of >=4096 HMAC rounds, run on-reactor before the body is parsed. Under
high-rate authenticated traffic this dominates CPU. A cluster serving a few
hundred HTTP-produce req/s per shard through PandaProxy saturated its reactor
core almost entirely in this path (~45% of on-CPU time in
validate_scram_credential), starving heartbeats.

This memoizes the derivation with a bounded per-shard cache, keyed by an HMAC
of (algorithm, password, salt, iterations) and holding only the derived stored
key (already in the credential store). It is purely functional, so a password
change re-salts and re-keys and entries never go stale.

The cache is gated by the scram_credential_cache_enabled cluster config and
ships disabled by default. Because it caches the result of password
validation in memory, enabling it by default is deferred pending security team
review; until then it is strictly opt-in and out-of-the-box behavior is
unchanged.

Fixes CORE-16829

Security considerations (for review)

Only the derived stored key is cached — the value already persisted in the
credential store; no plaintext or salted password is retained.
The cache key is an HMAC of (algorithm, password, salt, iterations) under a
random per-boot key, so a memory dump exposes no offline-crackable material
and keys can't be correlated across shards or reboots.
Every hit re-compares the cached stored key against the credential's live
stored key, so a rotated credential (re-salted, hence re-keyed) or a
cache-key collision fails closed. A false accept would require colliding both
the HMAC key and the SCRAM stored key at once — two independent SHA
collisions, one under a secret key — negligible and no weaker than SCRAM's
existing assumptions.
Hits (~0.6 µs) and misses (~0.4 ms) are timing-distinguishable, but both
correct and wrong passwords are cached, so a hit only reveals that an exact
(user, password) was tried recently — not whether it is correct. Correctness
still comes from the 200/401 response, and distinct guesses (a dictionary
attack) always miss. Caching failed validations also keeps bad-credential
floods cheap.

Microbench (`--config=release`, single core)

Repeat = same credential per call (steady-state auth); unique = fresh password
per call (cache-miss path).

Validation case	Disabled (default)	Enabled	Speedup
SHA-256, correct password (repeat)	391 µs	558 ns	~700x
SHA-256, wrong password (repeat)	392 µs	568 ns	~690x
SHA-512, correct password (repeat)	1.04 ms	554 ns	~1,880x
SHA-256, unique password (cache miss)	390 µs	396 µs	unchanged

End-to-end (single core, unbatched ~2 KiB HTTP produce)

Scenario	Disabled (default)	Enabled
Basic-auth produce	288 req/s, 3.43 ms/req	635 req/s, 1.56 ms/req
No-auth baseline (same path, no auth)	589 req/s, 1.68 ms/req	660 req/s, 1.50 ms/req
Basic-auth, 50% empty-body requests	336 req/s, 2.96 ms/req	1,354 req/s, 0.74 ms/req

The cache is disabled by default, so out-of-the-box throughput is the Disabled
column. Enabling it (scram_credential_cache_enabled=true) raises Basic-auth
throughput to match the no-auth baseline; toggling it back off returns to the
Disabled numbers, so the config gates the behavior cleanly in both directions.

Backports Required

Release Notes

none

Measures the per-call cost of the SCRAM password check that HTTP Basic auth runs on every pandaproxy/schema registry/admin request. The salted password is rederived per request (4096 HMAC rounds), which saturated pandaproxy CPU under unbatched HTTP produce in a production incident: ~473us per SHA-256 validation, so a few thousand requests per second exhaust a core before any request work happens. Covers the wrong password (reject) path, which costs the same as an accept, SHA-512 (~3x SHA-256), and a unique-password case that always pays the full derivation.

Exposes the password -> stored_key derivation as a reusable step so a subsequent commit can memoize it.

Copilot

Pull request overview

This PR introduces a per-shard memoization layer for SCRAM password validation to eliminate repeated PBKDF2-style derivations on hot authentication paths (HTTP Basic auth and SASL/PLAIN), controlled via a new cluster configuration kill switch.

Changes:

Add security::scram_credential_cache and integrate it into validate_scram_credential with a thread_local per-shard cache gated by scram_credential_cache_enabled.
Refactor SCRAM algorithm code to expose derive_stored_key() and reuse it in password validation.
Add unit tests and an rpbench benchmark covering cached vs uncached behavior and correctness.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/v/security/tests/scram_credential_cache_test.cc	New unit tests validating cache hit/miss behavior, key composition, eviction bounds, and cached credential validation.
src/v/security/tests/scram_credential_bench.cc	New perf benchmark exercising repeated/unique password validation cases to quantify caching impact.
src/v/security/tests/scram_algorithm_test.cc	Adds a test ensuring `derive_stored_key()` matches stored keys produced by credential creation.
src/v/security/tests/BUILD	Wires new test and benchmark targets into the build.
src/v/security/scram_credential_cache.h	Declares the per-shard SCRAM derivation cache and its keying/metrics interface.
src/v/security/scram_credential_cache.cc	Implements HMAC-keyed cache entries backed by `utils::chunked_kv_cache`.
src/v/security/scram_authenticator.h	Documents cached validation behavior and introduces a `detail::validate_scram_credential` overload for caller-provided caches.
src/v/security/scram_authenticator.cc	Implements cached derivation path and config-gated use of a per-thread cache.
src/v/security/scram_algorithm.h	Adds `derive_stored_key()` and refactors `validate_password()` to use it.
src/v/security/BUILD	Adds the new cache implementation to the security library and links required deps.
src/v/config/configuration.h	Declares the new `scram_credential_cache_enabled` configuration property.
src/v/config/configuration.cc	Defines the new configuration property and its description/default.

nguyen-andrew · 2026-07-08T22:40:04Z

Force push to address Copilot comments.

vbotbuildovich · 2026-07-09T00:18:47Z

CI test results

test results on build#86904

test_status	test_class	test_method	test_arguments	test_kind	job_url	passed	reason	test_history
FLAKY(PASS)	ShadowLinkTopicFailoverTests	test_producer_ids_failover	{"storage_mode": "tiered_cloud"}	integration	https://buildkite.com/redpanda/redpanda/builds/86904#019f43fb-b592-4841-ae9b-a6f7097397b7	19/21	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0079, p0=0.1462, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3917, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ShadowLinkTopicFailoverTests&test_method=test_producer_ids_failover

test results on build#86936

test_status	test_class	test_method	test_arguments	test_kind	job_url	passed	reason	test_history
FLAKY(PASS)	NodeWiseRecoveryTest	test_recovery_local_data_missing	{"wait_for_final_manifest_uploads": true}	integration	https://buildkite.com/redpanda/redpanda/builds/86936#019f48a5-8cb9-4fde-94e4-69a6b46cdedd	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0555, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1574, p1=0.1805, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=NodeWiseRecoveryTest&test_method=test_recovery_local_data_missing
FLAKY(FAIL)	ShadowLinkingRandomOpsTest	test_node_operations	{"failures": true, "workload_set": "cloud_combos"}	integration	https://buildkite.com/redpanda/redpanda/builds/86936#019f48a5-8cbc-468b-9578-e4647d82e748	9/11	Test FAILS after retries.Significant increase in flaky rate(baseline=0.0008, p0=0.0082, reject_threshold=0.0100)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ShadowLinkingRandomOpsTest&test_method=test_node_operations

test results on build#86985

test_status	test_class	test_method	test_arguments	test_kind	job_url	passed	reason	test_history
FLAKY(PASS)	Datalake3rdPartyMaintenanceTest	test_e2e_basic	{"catalog_type": "rest_jdbc", "cloud_storage_type": 1, "query_engine": "spark"}	integration	https://buildkite.com/redpanda/redpanda/builds/86985#019f4f11-0ec8-4c09-b66f-974b65ac4f20	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0000, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=Datalake3rdPartyMaintenanceTest&test_method=test_e2e_basic
FLAKY(PASS)	NodeWiseRecoveryTest	test_recovery_local_data_missing	{"wait_for_final_manifest_uploads": true}	integration	https://buildkite.com/redpanda/redpanda/builds/86985#019f4f11-f890-469c-a326-8bdc8c40554b	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0531, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1511, p1=0.1945, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=NodeWiseRecoveryTest&test_method=test_recovery_local_data_missing

pgellert · 2026-07-09T08:44:26Z

/microbench

vbotbuildovich · 2026-07-09T11:33:18Z

Performance change detected in https://buildkite.com/redpanda/redpanda/builds/86918#019f4614-84b6-4266-9e37-2288a79527c5:

Performance changes detected in 10 tests
cloud_storage_rpbench.cstore_bench.cs_iteration_recompute_end_test_1000: inst -> -0.05pct
wasm_transform_rpbench.WasmBenchTest_BatchSize1_RecordSize1_KiB.IdentityTransform: inst -> -0.04pct
async_algorithm_rpbench.algo_bench.sync_std_for_each_big: allocs -> +infpct
role_store_bench_rpbench.role_store_bench.put_role: inst -> +0.06pct
authorizer_bench_rpbench.authorizer_bench.describe_10k_immediate_deny: inst -> +0.75pct
authorizer_bench_rpbench.authorizer_bench.describe_10k_prefix_acls: inst -> +0.50pct
authorizer_bench_rpbench.authorizer_bench.describe_10k_prefix_acls_allowed: inst -> +0.60pct
authorizer_bench_rpbench.authorizer_bench.describe_10k_prefix_acls_no_match: inst -> +1.78pct
authorizer_bench_rpbench.authorizer_bench.describe_10k_prefix_acls_two_groups: inst -> +0.26pct
authorizer_bench_rpbench.authorizer_bench.describe_literal_grants_10k: inst -> -0.87pct

See https://redpandadata.atlassian.net/wiki/x/LQAqLg for docs

nguyen-andrew · 2026-07-09T20:21:01Z

Force push to change default behavior to disabled.

vbotbuildovich · 2026-07-09T21:55:33Z

Retry command for Build#86936

please wait until all jobs are finished before running the slash command

/ci-repeat 1
skip-redpanda-build
skip-units
skip-rebase
tests/rptest/tests/shadow_linking_rnot_test.py::ShadowLinkingRandomOpsTest.test_node_operations@{"failures":true,"workload_set":"cloud_combos"}

dotnwat

good stuff. just some questions

dotnwat · 2026-07-10T20:34:39Z

+/// changes the cache key.
+class scram_credential_cache {
+public:
+    static constexpr size_t default_capacity = 1024;


should be a configuration option?

Could be, but not sure if it'd be useful or just a config option that's ignored and never gets changed. 1024 seems like a reasonable default, something like ~256 KB per shard. But happy to add it if you think we should.

dotnwat · 2026-07-10T20:35:40Z

+    stats_t stats() const;
+
+private:
+    static constexpr size_t key_digest_size = 32;


is this a duplicate of the one at the top of the file?

Good catch, folded them together.

dotnwat · 2026-07-10T20:41:58Z

+  const credential_password& password,
+  bytes_view salt,
+  int iterations) const {
+    hmac_sha256 mac(_digest_key);


can you explain why we use hmac to build the key (for example as opposed to feeding all the parameters and the secure digest through sha256)?

So from what I read, directly hashing sha256(secret ‖ params) has a length-extension weakness and HMAC is supposed to be the standard way to key a hash with a secret. And the secret is there so the keys aren't plain password hashes.

dotnwat · 2026-07-10T20:45:57Z

+    auto valid = stored_key == cred.stored_key();
+    if (cache != nullptr) {
+        cache->put(
+          mech,


if mech is not used to compute the stored key, why is it part of the cache key?

Right, the mech value itself isn't used in the computation, the scram template parameter is, but they're supposed to be related. I'll update it to derive mech from the template instead of passing it as well.

dotnwat · 2026-07-10T20:47:00Z

+
+namespace detail {
+
 std::optional<std::string_view> validate_scram_credential(


should we restrict to the rest proxy authentication? kafka, admin, etc... use scram but likely don't have any performance concerns (e.g. kafka is per connection auth).

Looks like the only callers to this validate_scram_credential (and users of this cache) are the HTTP Basic surfaces (pandaproxy, schema registry, admin) that re-auth on every request plus Kafka SASL/PLAIN (but only per-connection). Kafka SASL/SCRAM goes down a different path so it doesn't use it. Since they all flow into here, I think it's easier/cleaner to leave it than to pull it out/restrict it to just panda proxy, but that's really my only reason - we can restrict it if you think there's a good reason to.

Per-shard memoization of SCRAM password derivations, keyed by an HMAC (random per-instance key) of (mechanism, password, salt, iterations) so plaintext passwords are not retained, holding only the derived stored key: the same datum already persisted in the credential store. The mapping is purely functional, so entries never go stale: updating a credential generates a fresh salt, which changes the cache key. Built on the S3-FIFO chunked_kv_cache, whose scan resistance keeps one-shot wrong-password entries from displacing hot legitimate ones.

validate_scram_credential now consults a shard-local scram_credential_cache, skipping the salted password derivation for repeat authentications. HTTP Basic auth (pandaproxy, schema registry, admin API) and SASL/PLAIN pay that derivation on every request; in production, unbatched HTTP produce through pandaproxy at a few hundred requests per second saturated a core on it. Cache hits skip the derivation entirely; the uncached path is unchanged. The scram_credential_cache_enabled cluster config (default true) is an operational kill switch.

nguyen-andrew · 2026-07-11T02:15:46Z

Force push to address PR comments.

nguyen-andrew added 2 commits July 8, 2026 19:42

security: extract derive_stored_key from validate_password

b29abbc

Exposes the password -> stored_key derivation as a reusable step so a subsequent commit can memoize it.

Copilot AI review requested due to automatic review settings July 8, 2026 21:26

nguyen-andrew requested a review from a team as a code owner July 8, 2026 21:26

github-actions Bot added area/build area/redpanda labels Jul 8, 2026

Copilot started reviewing on behalf of nguyen-andrew July 8, 2026 21:27 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Comment thread src/v/security/scram_credential_cache.cc Outdated

Comment thread src/v/security/scram_credential_cache.h Outdated

nguyen-andrew force-pushed the scram-memo branch from a1eec85 to a269df2 Compare July 8, 2026 22:39

nguyen-andrew self-assigned this Jul 8, 2026

nguyen-andrew requested review from bartoszpiekny-redpanda, dotnwat, mnajda-redpanda and pgellert July 9, 2026 01:16

nguyen-andrew requested a review from ballard26 July 9, 2026 17:22

nguyen-andrew force-pushed the scram-memo branch from a269df2 to a81d011 Compare July 9, 2026 20:19

nguyen-andrew changed the title ~~security: cache SCRAM password validation~~ security: cache derived SCRAM stored key Jul 9, 2026

dotnwat reviewed Jul 10, 2026

View reviewed changes

nguyen-andrew added 2 commits July 11, 2026 02:07

nguyen-andrew force-pushed the scram-memo branch from a81d011 to cb63f4f Compare July 11, 2026 02:15


		namespace detail {

		std::optional<std::string_view> validate_scram_credential(

Uh oh!

Conversation

nguyen-andrew commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Security considerations (for review)

Microbench (--config=release, single core)

End-to-end (single core, unbatched ~2 KiB HTTP produce)

Backports Required

Release Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

nguyen-andrew commented Jul 8, 2026

Uh oh!

vbotbuildovich commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI test results

Uh oh!

pgellert commented Jul 9, 2026

Uh oh!

vbotbuildovich commented Jul 9, 2026

Uh oh!

nguyen-andrew commented Jul 9, 2026

Uh oh!

vbotbuildovich commented Jul 9, 2026

Retry command for Build#86936

Uh oh!

dotnwat left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nguyen-andrew commented Jul 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

nguyen-andrew commented Jul 8, 2026 •

edited

Loading

Microbench (`--config=release`, single core)

vbotbuildovich commented Jul 9, 2026 •

edited

Loading